Add snailtracer benchmark
#2
Conversation
|
Huh... I completely forgot about the existence of pyevm lol. Lemme see if I can get this benchmark running there. |
|
Let's just say... slower. Average of ~39028ms per run. That's about 345x slower, wow. I tried to get the benchmark as barebones as possible, I think I actually trimmed it more than the benchmark for pyrevm, if you can spot anything that would be causing such a huge slowdown, let me know: https://gist.github.com/ziyadedher/d07610ad172cf25fff7d0558c522304d But it does seem like the majority of time is spent evaluating opcodes according to the profile below. |
|
Suspect there's a lot of overhead due to pyevm doing header validation, vs just being a thin interpreter |
|
I did some pruning and (1) verified no header validation is being done and (2) no state is being committed in the critical paths for the pyevm benchmarks. Traced the code line-by-line and AFAICT it's just straight up transaction execution, running opcodes. I even got rid of the profiler in case that had some overhead. I end up with ~15693ms per run of this benchmark compared to ~115ms for pyrevm under the same harness. That's ~135x faster. Incredible. I don't see any reason to doubt this right now, but happy to be proved wrong. |
|
Wow that's wild, thank you for running this. Any way to reproduce the result with your pruned pyevm? |
|
Removed the profiler from the I just mocked out the StateAPI, basically commenting out all the DB functionality in In the gist, here's how the control flow goes. There's a lot of weird overriding and inheritance going on that makes it hard to manually trace:
If someone wants to repro, I'd encourage mocking out the StateAPI to start, but also challenging some of the assumptions I made here about what's expensive and what's not. But the profiler output seems to corroborate those assumptions. Here's some profiler output. Note the walltimes are not true-to-life since the profiler adds significant overhead, but it gives a sense. |
|
This is great. Appreciate you taking the time to note everything down. |
|
@SamWilsn recommended trying w/ PyPy, it's a lot faster! ~4290ms per run w/ PyPy compared to ~15693ms w/ CPython. So Below is a summary of results so far. This is on a i7-7700K CPU @ 4.20GHz.
Below is a profile from the PyPy run, again walltimes are misleading. |
|
BTW I threw together https://github.com/ziyadedher/evm-bench, here's how pyrevm compares on a few different benchmarks against friends. I'll probably get geth in there at some point too.
|
Note: this PR includes #1 cuz it made it easier
Did some benchmarking using snailtracer -- results are pretty good IMO:
~50ms using revm, ~113ms using pyrevm, eh not bad. I expected it to be a bit closer, but I imagine a good portion of this is Python FFI overhead. Haven't dug into it too much.
This isn't a perfect apples-to-apples comparison, the revm benchmark modifies the transaction context before starting the benchmark, this benchmark constructions the environment during the benchmark. With a little more effort we could probably remove any additional overhead due to that, but I don't think it's that big of a deal.
Baseline results using the
snailtracerbenchmark from revm:Results from this benchmark: